As background, in the remote sensing/aerial imaging industry, imagery is used to capture views of a geographic area and to be able to measure objects and structures within the images as well as to be able to determine geographic locations of points within the image. These are generally referred to as “geo-referenced images” and come in two basic categories:
Captured Imagery—these images have the appearance they were captured by the camera or sensor employed.
Projected Imagery—these images have been processed and converted such that they confirm to a mathematical projection.
All imagery starts as captured imagery, but as most software cannot geo-reference captured imagery, that imagery is then reprocessed to create the projected imagery. The most common form of projected imagery is the ortho-rectified image. This process aligns the image to an orthogonal or rectilinear grid (composed of rectangles). The input image used to create an ortho-rectified image is a vertical or nadir image—that is, an image captured with the camera pointing straight down. It is often quite desirable to combine multiple images into a larger composite image such that the image covers a larger geographic area on the ground. The most common form of this composite image is the “ortho-mosaic image” which is an image created from a series of overlapping or adjacent nadir images that are mathematically combined into a single ortho-rectified image.
Because the rectilinear grids used for the ortho-mosaic are generally the same grids used for creating maps, the ortho-mosaic images bear a striking similarity to maps and as such, are generally very easy to use from a direction and orientation standpoint. However, because the images are captured looking straight down, most people have difficulty determining what they are seeing since people rarely see the world that way. There is an entire discipline dedicated to working with vertical or nadir imagery known as photo interpretation which teaches people how to read subtle clues in the image to make a determination of what the object they are seeing might be.
It is for this reason that Pictometry created fully geo-referenced oblique imagery. Like ortho-rectified nadir images, these images have the ability to support measurements, determine locations, measure heights, and overlay annotations and GIS data. However, they are captured at an oblique angle so that they capture not only the top of structures and objects, but also the sides as well. This is a much more natural view that allows anyone to use aerial imagery—it eliminates the need to train in photo interpretation or have years of experience in order to make confident assessments regarding the content of the imagery. U.S. Pat. No. 5,247,356 describes a preferred embodiment of their initial oblique image capture system. Since then, significant improvements have been made to the system, still based on the '356 patent. The current system is capable of capturing five views simultaneously: four oblique views, each oriented roughly along the four cardinal directions, plus a nadir view capturing the area directly below the aircraft. All the images captured by this system are full geo-referenced in real-time and then can be post-processed to increase the accuracy of the geo-referencing.
In producing the geo-referenced aerial images, hardware and software systems designed for geo-referencing airborne sensor data exist and are identified herein as a “POS”, i.e., a position and orientation system. For example, a system produced by Applanix Corporation of Richmond Hill, Ontario, Canada and sold under the trademark “POS AV” provides a hardware and software system for directly geo-referencing sensor data. Direct Georeferencing is the direct measurement of sensor position and orientation (also known as the exterior orientation parameters), without the need for additional ground information over the project area. These parameters allow data from the airborne sensor to be georeferenced to the Earth or local mapping frame. Examples of airborne sensors include: digital aerial cameras, multi-spectral or hyper-spectral scanners, SAR, or LIDAR.
The POS system, such as the POS AV system, is mounted on a moving platform, such as an airplane, such that it is held firmly in place relative to the sensors for which it is measuring position and orientation. By doing such, a single POS system can record the position and orientation of multiple sensors. In addition, if the POS system incorporates GPS or GLONASS, an antenna is mounted on the platform such that it has a clear view of the sky in order to receive signals from a satellite constellation. If the system incorporates an angular measurement capability, such as a fiber optic gyro, mechanical gyro, mechanical tilt sensor, or magnetometer, these systems must be mounted in a manner that holds them firmly in place relative to the sensors for which they are measuring orientation. If measurements must be taken more frequently than the actual measured positions and orientations then a highly accurate clock is incorporated and a means to record the precise clock time of any sensor capture event is integrated. For instance, with a shutter based camera, an electrical signal can be sent at the time the shutter is fully open triggering the POS system to record the precise time on the clock for that sensor capture event.
In the past, the images and the time and position data were stored on hard drives in the airplane and were post processed and made available to users after the airplane landed. This process could take days or even weeks before geo-referenced images were made available to users. Normally, these time periods are within the relevant time-frame. However, after a disaster occurs, this is not necessarily the case.
In the past, post-disaster metric aerial oblique imagery has been captured and processed and is very useful to first responders and to those responsible for rebuilding. This is especially true for hurricanes and floods where the oblique imagery shows the height the water has reached up the sides of buildings—something difficult to ascertain from traditional straight down orthogonal imagery.
During the aftermath of Hurricane Katrina, a new need was discovered: the need to determine the immediate extent of the flooding and damage and relay that to the first responders in the field. While Hurricane Katrina left a large swath of destruction, some areas were more devastated than others. What would have been extremely useful was to conduct an overflight, transmit that data directly to the ground, allow first responder specialists to look at the imagery and select the most affected areas or other critical pieces of infrastructure such as evacuation routes that might possibly be blocked, and have the aircraft capture those areas in more detail.
The presently disclosed and claimed invention was created in response to that need. The work was driven by the Department of Homeland Security (DHS) which asked for a system that could perform real-time georeferencing of aerial imagery and then transmit the images to the ground for display in a Geographic Information System (GIS). The patent owner, i.e., Pictometry, was awarded a Small Business Innovation Research (SBIR) grant to create such a system for DHS and FEMA—the Federal Emergency Management Administration. The presently disclosed and claimed inventive concepts go beyond the needs and specifications of the SBIR and adds the ability to do these tasks with sensor data such as but not limited to metric oblique imagery, as well as straight down orthogonal imagery.
Satellite image capture systems exist, but while they have the ability to transmit from the sensor to the ground, this does not immediately get the information into the first responders in the field. First, the satellite cannot loiter over an area, e.g., fly multiple contiguous flight paths—it must maintain its orbit and therefore only comes by a particular geographic region every so often. Even with the ability to task the sensors on the satellite that generally only widens the window of opportunity over the target or increases the frequency over the target—it still does not allow it to loiter about a predetermined ground area. Second, even if a satellite image capture system could loiter, because satellites fly so high over the earth, any cloud cover will obstruct their view of the ground. Since there is typically a lot of cloud cover after weather related disasters, such as hurricanes, floods, and tornadoes, this presents a serious problem, further compounded by the satellites inability to loiter. Third, many satellites download the data in a batch format when they are passing over an appropriate receiving station, rather than downloading images in real-time to a van or other ground station on site at the emergency response center. Fourth, most satellite data requires significant post-processing in order to put the images into a form that can be readily understood or used by the Geospatial Information Systems (GIS) and Computer Aided Dispatch (CAD) systems the first responders use during emergency response.
Traditional aerial image fliers do not provide the captured data directly into the hands of the first responders in the field in real-time for a variety of reasons. First, the data rates off the sensor are generally prohibitive for successfully transmitting data to the ground in real-time. Second, the imagery typically needs to be ortho-rectified in order to make it usable in GIS and CAD systems. Third, there was no known and available direct download systems in the industry capable of reliably downloading the data from the airplane to the ground. Fourth, the data is normally captured from directly overhead which is a view that first responders are not used to seeing. GIS experts typically take courses in photo interpretation in order to learn how to recognize structures and details from straight down imagery. Few first responders have had this education or the requisite experience.
With respect to the downloading of the captured data from an airplane to the ground, conventional methodologies include manually aiming a dish antenna in the general direction of a moving remote platform and then fine-tuning the aiming utilizing the signal strength of the incoming signal. This works acceptably for remote platforms such as airships that are hovering over a fixed location. However, this is often impractical or unreliable for communicating with a communication system carried by an airplane used to capture images with the aid of a flight plan in response to a disaster and which may be travelling more than 25 miles away from the dish antenna. Further, the conventional methodologies did not provide an automated method for reestablishing a connection or data synchronization problems after a drop-out of the high speed link.
Thus, there is a need for a system that can capture, process (e.g., develop and geo-reference) and download sensor data such as but not limited to metric oblique aerial images in real-time for use by first responders in response to a natural or man-made disaster. It is to such a system that the presently disclosed and claimed inventive concepts are directed.